Egress tunnel capture + agent identity#338
Egress tunnel capture + agent identity#338Nina Polshakova (npolshakova) wants to merge 9 commits into
Conversation
Nina Polshakova (npolshakova)
force-pushed
the
egress-tunnel
branch
2 times, most recently
from
9b63f65 to
e678eeb
Compare
| cleanupErr = errors.Join(cleanupErr, fmt.Errorf("while removing actor nftables rules: %w", err)) | ||
| slog.WarnContext(ctx, "Failed to remove actor nftables rules; continuing actor netns cleanup", slog.Any("err", err)) | ||
| } | ||
| if s.egressCapture != nil { |
There was a problem hiding this comment.
Is there a time we would want no egress capture?
There was a problem hiding this comment.
That's a good question- I think in general, egress capture should always be enabled but you can imagine scenarios where that doesn't make sense:
- No PEP egress is setup. In this case I think traffic should still be captured by the ateom proxy, but not forwarded to any PEP
- Flexibility for workloads using traffic we do not capture yet (UDP/QUIC, non-80/443 TCP if we don't want to do the classifier, etc.)
There was a problem hiding this comment.
Mike Morris (@mikemorris) This is kind of related to the discussion we had in slack. Like you brought up, if we're expecting all traffic to always egress through a PEP, then the PEP registration can be a first-class API. Are there cases though where we don't want the traffic to get captured/egress through a PEP?
Nina Polshakova (npolshakova)
force-pushed
the
egress-tunnel
branch
2 times, most recently
from
2e910e3 to
a360515
Compare
Nina Polshakova (npolshakova)
force-pushed
the
egress-tunnel
branch
from
d853537 to
18aad2f
Compare
|
I agree with this approach overall. However there are problems that will need to be addressed:
To improve the story with gateway pluggability, I propose to add support for PROXY protocol. This will make it easier to plug other gateways, like KAN or Envoy AI Gateway. This can be a substrate egress configuration option, which will make config more complex, and can be added as a followup (I volunteer myself for this if needed). Eventually and option for terminating TLS at the proxy and re-encrypting using mTLS with agent identity could be added as well. This will make it even easier to plug other gateways as PEPs. |
|
Eventually even the thin proxy should be pluggable. Operators will want telemetry, tracing, access logging from it. It will need to be able to load balance across a dynamically scaled fleet of PEPs, implement retries, health checking, etc. We do not want to recreate all this stuff in a custom go proxy. We can start with just a DNS name and go forwarder, but eventually it will need evolve and replicating functionality that already exists. |
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Nina Polshakova (npolshakova)
force-pushed
the
egress-tunnel
branch
from
18aad2f to
938b88b
Compare
Signed-off-by: npolshakova <nina.polshakova@solo.io>
4 participants
POC for #126
Based on design discussed in https://docs.google.com/document/d/1KmpIFu2gnqy9gp95wASgIo_vkJ_dA1DZckV8upET6bs/edit?usp=sharing
Summary:
This is a proof of concept egress capture path for actors. It introduces a reusable
internal/egresspackage that:Hostor TLS SNI, and opens a CONNECT-style tunnel to a resolved PEP address.The gVisor and microvm runtimes wire this into actor network setup by redirecting actor HTTP/HTTPS egress traffic to local capture ports. The Linux-only glue shared by both runtimes (SO_ORIGINAL_DST lookup, nftables redirect rules, identity construction) lives in
internal/ateomegressso the two runtimes stay in sync. Agentgateway is used as the receiving proxy to prove that captured actor traffic reaches the tunnel endpoint.Capture is enabled per actor rather than by a global switch: ate-api watches Gateways labeled
ate.dev/egress-pep, resolves the best-matching PEP for each actor at resume time (actor-specific > atespace-scoped > global), sends the PEP address down through atelet to ateom, and records the resolved address on the Actor status(egress_pep_address) so the binding is observable to the user.NOTE: Currently, an actor with no matching PEP resolves to an empty address and runs with capture off.
Notes:
x-ate-actor-idx-ate-atespacex-ate-actor-templatex-ate-actor-template-namespacex-ate-original-destinationx-ate-connect-authoritydocs/egress-capture.md). A first-class PEP binding/control plane is out of scope of this PR — agentgateway is just used to prove the proxy gets traffic from the tunnel.status.addresses(falling back to the<gateway>.<namespace>.svcconvention) and only considersProgrammed=TrueGateways.ateom.proto). TLS stays end-to-end; agentgateway only routes the encrypted bytes after CONNECT.